The statements in this section may serve as a background to help understand the invention and its application and uses, but may not constitute prior art.
Modern computing technology have brought in a new era of immersive experiences, with virtual reality, augmented reality, and other immersive environments that blur the boundary between physical and digital space. Whether playing a computer video game, watching a sports event, or exploring 3D models, immersion enhances the experience by making it engaging, interactive, and realistic, with images, sounds, and haptic feedbacks that establish the user's presence in an immersive three-dimensional (3D) environment. For example, 360-degree panorama videos taken with action cameras are readily available on video sharing websites such as YouTube for viewers to experience action adventures such as skydiving and car racing. Virtual reality and augmented reality have seen many uses in applications such as gaming, movies and television, architecture, and telepresence. Yet ground-up development of immersive media content for mass mainstream adoption is still non-trivial and expensive.
A critical bottleneck in immersive media content generation and distribution is the enormous amount of data that need to be generated and transmitted from a media content server to viewers to build a 360-degree environment with high resolution, high framerate, and low latency. For one, the full 360-degree horizontal field of view is twice the human visual field for binocular vision, while conventional displays occupy only a portion of the human visual field. For two, with dedicated head-mounted displays, at standard resolutions, individual pixels on the displays become noticeable since VR headsets place display screens or panels within an inch of the eye, making images blurry and small texts close to impossible to read. For three, at low or standard refresh rates, a user in motion may experience sensory conflict, a disparity between what is expected and what is seen on screen, thus triggering motion sickness. In short, improved visual fidelity and imperceptible delays are desirable for providing accurate and realistic immersive environments while also limiting or avoiding motion sickness. Furthermore, to make the immersive experience more authentic, dedicated headsets are preferred to be wireless, a design solution yet to become practical because of limitations on hardware performance and data throughput. Thus, tradeoffs among computing power, data transfer rates, processing capabilities, and user experience and comfort become extremely important when perfecting the immersive experience.
Within the computer video gaming universe, while development of VR-games is inherently challenging due to the reasons discussed above, converting existing video games and eSport platforms into their VR counterparts is also known for being very difficult. Traditional controls and user interfaces often do not work well in VR, while simulation sickness and latency also need to be taken into account. Moreover, 360-degree rendering and recording of an existing 3D game is generally too resource intensive for low-end platforms or user devices, where rendering, compressing, streaming, and even replaying of 360-degree videos necessitate high processing power, careful memory management, and bandwidth optimization. Furthermore, production-ready offline rendering systems can produce 360 panoramas for game replay in a VR environment, but such systems are generally incapable of live capture and streaming.
Therefore, in view of the aforementioned difficulties, there is an unsolved need to make it easy to capture, record, stream, replay, and broadcast live or off-line immersive media contents, including game plays of non-VR and VR video games in virtual reality environments. In addition, it would be an advancement in the state of the art of immersive media content generation and distribution to render with high visual fidelity while creating minimal delay and data transfer overheads. It is against this background that various embodiments of the present invention were developed.